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PREFACE 



The purpose of this questionnaire is to aid the interested profes- 
sional in describing, characterizing, and analyzing his game, model, or 
simulation. It is a first and, hopefully, useful step in the clarifica- 
tion of professional standards in the work on gaming and simulation. 
Furthermore, the questionnaire is designed so that it might also serve 
as a device for communicating and cataloging different games and simula- 
tions in a format that encourages easy interchange of information. 

The questionnaire format has been adopted for three purposes. 

(1) In the course of our ongoing investigations we expect to use this 
document as a questionnaire. (2) Stress in design has been to produce 
a categorization scheme for the description and classification of games 
and simulation in general. The goals are to help establish professional 
standards and to explore the possibilities of developing a reasonably 
good classification and consistent description that covers many games of 
different varieties. (3) It is our belief that the compiling of a large, 
consistent sample of many games and simulations for the purposes of anal- 
ysis, evaluation, information interchange, and the construction of pro- 
fessional standards is overdue. The work involved in doing so is both 
large and onerous. The handling of large quantities of data calls for 
at least partially computerized procedures. The format we are pre- 
senting here was designed with this type of data processing in mind. 



Any views expressed in this paper are those of the author. They 
should not be interpreted as reflecting the views of The Rand Corporation 
or the official opinion or policy of any of its governmental or private 
research sponsors. Papers are reproduced by The Rand Corporation as a 
courtesy to members of its staff. 
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This questionnaire had as an original purpose the description and 
classification of games in general. Because the preponderance of all 
available resources for models, simulations, and games is spent by the 
Defense Department and its various derivates, we have redesigned and 
reorganized the instrument to reflect these activities better. Some 
questions were reworded, some were added, others were discarded. At 
this point in the process, separate questionnaires for computer, all 
machine models, and for man-machine and manual games were constructed. 

These modified questionnaires were applied to ten representative 
DOD models, simulations, and games. Several Rand Corporation games 
were also sampled. This "shakedown 1 * prompted another round of evalu- 
ation and redesign. The present version of a single questionnaire is 
the result. Although the emphasis on DOD is evident, we believe that 
a hard core of generally applicable information remains. 

Questionnaire conceptualization and design is a potentially diffi- 
cult undertaking. This particular questionnaire covers a complex, diverse, 
changing, and specialized body of knowledge and practice; consequently, 
the difficulties found in its construction have been formidable. For 
instance, even at the most primitive definitional level there is little 
but cloying disagreement. What is a model? What difference is there 
between a model and a simulation? What is a game and when is it not a 
model, and vice versa? Indeed the semantic game presently appears to 
take precedence and to substitute for the real game all too frequently. 
Other indicators of an unsettled but emerging professionalism abound. 

What this means is that construction of a questionnaire such as this is 
hard work; and no matter what results, it will have shortcomings, prob- 
lems, and probably more than a few errors. We acknowledge the weaknesses 
and welcome comments, suggestions, and corrections from others concerned 
with gaming and simulation. 

Parts I, II, and IV are designed for all types of models, games, and 
simulations. Manual and man-machine activities necessitated the addi- 
tional questions in Part III as well. Further work is in progress in the 
categorization of questions more specifically aimed at the uses of gaming 
and simulation for teaching and for experimental purposes. We recognize 
that this questionnaire cannot be regarded as adequate in providing a 
means of analysis if the two major uses of a game or simulation are teach- 
ing or research alone. 
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PURPOSES OF THE QUESTIONNAIRE 



1. To give an overall quick picture of the purposes, use, benefits, 
and costs of DOD activities in: 

Gaining 

Simulation 

and the Use of Models 

2. To provide an initial description and classification scheme for 
a professional catalog for games, simulations, and models for 
use in: 



(i) Quick professional interchange of 

information on the characteristics of 
models . 

(ii) Aid in construction and estimation of model 
characteristics for new model builders. 

(iii) Aid in the evaluation of previous and current 
activities . 



\ 




8 



2 



INTRODUCTION 



The state of the profession is such that no clear 
agreement on fundamental terms exists in all of 
the various activities using models, simulations, 
or games. Acknowledging this fact, we advance 
the following definitions to provide guides to 
indicate the type of work this questionnaire covers. 

TERMINOLOGY 



Gaming: A gaming exercise employs human beings acting as them- 

v selves or playing simulated roles in an environment 

which is either actual or simulated. The players may 
be experimental subjects or participants in an exercise 
being run for teaching, operational, training, planning, 
or other purposes. This questionnaire is designed to 
describe games constructed for these various purposes. 

War Gaming: One of the major applications of simulation is war gaming. 

A war game is defined by the Department of Defense as a 
simulation of a military operation involving two or more 
opposing forces and using rules, data, and procedures 
designed to depict an actual or assumed real-life situa- 
tion. It is primarily a technique used to study problems 
of military planning organization, tactics, and strategy. 

A war game can be conducted to cover the entire spectrum 
of war, i.e., politico-military crises, general war, 
or limited war. The game may be based on hypothetical 
situations, real-world crises, or current operational 
plans. Some games are designed for joint operations by 
two or more military services, some are for use by a single 
service, and others may be used by individual Army field 
commanders or even by division or battalion commanders. 

The level of command at which the game is to be played, 
of course, influences the type of units to be represented 
and the scope of operations to be conducted. 

There are three types of war games in common use today: 
the training game, the operational game, and the research 
game. The training game is the least complex and is de- 
signed to provide the participants with decisionmaking 
opportunities similar to those that may be experienced 
in combat. The operational game deals with current 
organizations, equipment, and tactics. It is more complex 
than the training game, uses inputs that are based on 
known quantities, and is used to test operational plans. 

The research game, which is the most complex of the 
three types of games, requires careful preparation to achieve 
maximum objectivity and usually is designed to study tactical 
or strategic problems in a future time frame. 
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Simulation: 



O 

ERIC 



A war game can be accomplished manually, can be computer- 
assisted, or can be wholly computerized. Manual games 
are played using symbols, pins, or pieces to represent 
forces, weapons, and targets on maps, mapboards, and 
terrain models. A computer-assisted game is a manual 
game using computerized models, which free the control 
group from many repetitive and time-consuming computations. 

Computer games are based on predetermined procedures. 

All simulation of conflict is done by the computer in 
accordance with the detailed instructions contained in 
the computer program. The primary advantage of computer 
gaming is that the same situation can be simulated many 
times under differing conditions, to observe variability 
of results. A computer war game requires the use of a 
war game model (i.e., computer program), which contains 
all the rules, procedures, and logic required to conduct 
the game. 

Simulation involves the representation of a system or 
organism by another system or model, which is designed to 
have a relevant behavioral similarity with the original 
system. Games utilize a sxmulated environment or simu- 
lated roles for the players or both. In general, all games 
are simulations. However, it is not particularly useful 
to use the reverse categorization. In other words, not 
all simulations are usefully regarded as games. Computer 
simulations stimulating conflict or cooperation (such as 
completely computerized battle models) are usefully con- 
sidered as games. Possibly, so are some logistic or resource 
allocation models where the single (automated or live player) 
team may be regarded as struggling against a statistical or 
strategic opponent called "Nature." The borderline is not hard 
and fast; however, we would expect to leave out a straight 
industrial production scheduling machine simulation from our 
category, for example. 



Computer simulation is an analytical technique which 
involves the use of mathematical and logical models to 
represent the study and behavior of real-world or hypo- 
thetical events, processes, or systems over extended 
periods of time. 

Simulation provides the means for gaining experience and 
for making and correcting errors without incurring the 
costs or risks of actual application. It offers opportu- 
nities to test theories and proposed modificiations in 
systems or processes; to study organizations and structures; 
to probe past, present, and future events; and to utilize 
forces that are difficult or impracticable to mobilize. 

10 
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Simulation therefore is of value both as an educational 
device and as a means of discovering improved methods. 

> 

Simulation should be used when (1) it is either impos- 
sible or extremely costly to observe certain processes in 
the real world, (2) the observed system is too complex 
to be described by a set of mathematical equations, 

(3) no straightforward analytical technique exists for 
solution of appropriate mathematical equations , and 

(4) it is either impossible or very costly to obtain 

data for the more complicated mathematical models describing 
a system. 

On the other hand, simulation should not be used when 
(1) simpler techniques exist, (2) data are inadequate, 

(3) objectives are not clear, (4) there are short-term 
deadlines, or (5) the problems are minor. 

Contract Studies 

& Analyses: The words n studies and analyses, " as used in this 

questionnaire, refer to those studies and analyses 
done by contract or by grant and which deal with the 
systematic and critical examinations of various subjects. 
Studies and analyses often require advanced analytical 
techniques to integrate a variety of factors and to 
evaluate data. Their purpose is to provide greater 
understanding of alternative organizations, tactics, 
doctrines, policies, strategies, procedures, systems, 
and programs;. 

i 

Model: Conducting a computer or computer-ass is ted war game or 

related simulation requires the use of a computerized 
model. As used in this context, a model is a document 
or program containing all rules, methodology, techniques 
procedures and logic required to simulate or approximate 
reality. A computerized model is a computer program or 
series of programs, designed to simulate the logic of 
actions or interactions of an environment or a context 
and provide the results to player personnel for subsequent 
analysis. 

MSG: To avoid the ambiguities of "local 11 definitional usage 

peculiarities, we have elected to use the shorthand 
version "MSG" to stand for model/simulation/game in the 
remainder of the questionnaire. : This, we hope, will 
facilitate present communications in light of the still 
unresolved, serious definitional problems. The respondent 
is encouraged to substitute for himself whatever local 
term seems appropriate when he encounters the "MSG" label. 
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LABELS/ADMINISTRATION 
(For Office Use Only) 



1-0 INTERNAL ID TAG : ( 



2-0 


PUBLICATION ID 


NUMBER: 


( 




) 


2-1 


PUBLICATION ID 


NUMBER: 


( 




) 


2-2 


PUBLICATION ID 


NUMEER: 


( 




) 


2-3 


PUBLICATION ID 


NUMBER: 


( 




) 


2-4 


PUBLICATION ID 


NUMBER: 


( 




) 



3-0 


ABSTPACTED : 


NO 


3-1 


ABSTRACTED: 


YES 


3-2 


A * rimrin « 

lju * 


nr* -r m t- nttn 
UUi iyujj 


3-3 


ABSTRACTED: 


AUTHOR ABSTRACT 
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INSTRUCTIONS 



In filling in this questionnaire's many questions as possible 
have been designed to be answered by marking one or more relevant 
categories in each question (e.g. > circling or marking a number). 

This should help to increase the speed with which the questions can 
be answered. Furthermore* it is easier to use computer assistance to 
process questions of this type. However, in many instances the words 
used for the categories do not quite reflect the properties of the 
game, simulation, or study being characterized. 

The respondents are ?:equested to mark the most relevant categories 9 
then to make any written commentary modifying the answer or suggesting 
an improvement of the question. 

Notes are provided on pages opposite the questions. 

Use one questionnaire for each man-machine game or simulation, 
machine (or analogue) simulation, or study. Parts I, II, and IV of 
this questionnaire are designed to be answered for all MSGs. Part 
III is added to accommodate the particular characteristics of man- 
machine and manual games . 

Note #1: Please note the amount of time taken to fill in each 

questionnaire and return to Question ill to record this information 
upon completion of a questionnaire. 

Note ill: To the right of each question there is a three-interval 

confidence scale. If you are certain of your answer mark an "X" 
at the extreme right on the scale. Please mark your confidence 
level for aZZ questions where the scale is indicated. 

Low Middle High 

I 1 i l 

Confidence Lrevel 

On a scale of 0-1 your answers will be interpreted as follows: 



Virtually certain 


.9-1 


High 


.7 - .9 


Middle 


.3 - .6 


Low 


0 - .3 
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If you wish you may use the scale in either of two ways, by using 
an X, 

l 1 * X 1 

or by putting in a number. 



"Certain" should be recorded as follows : 






Models and Simulations 

The word "model" is sometimes used to refer to a program that 
might be called a general purpose subroutine in the sense that it 
together with other models, may be run together in a simulation. Thus 
many simulations may be specific f, one shot” configurations of models 
operated only to answer a specific question. Sometimes a large scale 
program may be called a simulation and is used more than once with 
different inputs or even with models or subroutines add^d. If you 
feel that an important technical distinction concerning your MSG has 
not been made, comment accordingly and note where it might influence 
an answer. 

Builders and Users * 

The first section of this questionnaire is oriented more toward 
those who use the outputs from an MSG applied to a particular problem 
or study than to those who designed or built the MSG. A user who is 
reasonably well acquainted with the technical aspects of the work may 
be able to answer the second section of the questionnaire. However, if 
it appears necessary to have another individual or group answer the 
second part, this should be done and the second group should be 
identified. 
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PART I 



BASIC INFORMATION ON PURPOSES, USE, BENEFITS AND COSTS 




-f tr 

.a.*.} 
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*1-0 QUESTIONNAIRE TIME ( ) 

s k 

[Those Questions marked are of particular importance and 
therefore extra care and accuracy are called for.] 



If 2. Simulation/Study /Mo del and Author Name Tag: This calls for the 

name of the first or primary effort, list name of the agency or 
firm (s) that built the game, simulation or model-MSG. , and the name 
of the two or three major authors or designers. 

The agency or authority is the official group ordering the work. 
The sponsor calls for the name of the officer or official responsible 
for ordering the work. 



*2-0 SIMULATION /STUDY/MODEL NAME ( ) 

DESIGNER BUILDING AGENCY OR FIRM ( 

■ ■ ) 

AUTHOR(S) NAMES (S) ( ) 

AGENCY OR AUTHORITY (IES) ( ) 

( ) 

SPONSOR(S) OF WORK: NAME(S)( ) 

( . ) 



* 

3-0 CATEGORY : 
3-1 CATEGORY : 
3-2 CATEGORY : 
3-3 CATEGORY : 
3-4 CATEGORY : 
3-5 CATEGORY : 



MODEL 

SIMULATION 
MAN-MACHINE GAME 
MANUAL GAME 

MATHEMATICAL ANALYSIS OR STUDY 

OTHER ( ) Specify 
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H. Respondent's Role: [Describe your role with respect to this 

game or simulation, e.g., participant/player/funder/user/designer/ 
implementor/care taker, etc.] 



4-0 


RESPONDENT'S 


ROLE: 


FUNDER OR SPONSOR 


4-1 


RESPONDENT'S 


ROLE: 


USER 


4-2 


RESPONDENT'S 


ROLE: 


DESIGNER OR BUILDER 


4-3 


RESPONDENT ' S 


ROLE: 


PLAYER 


4-4 


RESPONDENT'S 


ROLE: 


CARETAKER 


4-5 


RESPONDENT'S 


ROLE: 


CONTROL TEAM OR REFEREE 


4-6 


RESPONDENT'S 


ROLE: 


OTHER ( 



Specify 



[COMMENT: Give or Attach a Brief Job Description] 



5-0 LENGTH OF TIME YOU HAVE BEEN ACQUAINTED WITH 

THIS WORK ( ) 



6-0 IF LESS THAN 3 TEARS y LIST LAST TWO ASSIGNMENTS ( 

.)' 



*7-0 RESPONDENT’S INFORMATION: NAME ( ) 

RANK, TITLE AND POSITION: ( ) 

( ) 



SPECIAL QUALIFICATIONS AND/OR EDUCATION RELEVANT TO THIS 
TASK: (Describe) 
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8-0 MAJOR STATED GENERAL PURPOSE OF YOUR MODEL, GAME, OR SIMULATION: 

[If you were given a few minutes to describe the project to a 
senior official, what could you say? (This could be one to two 
double-spaced, typed pages, or if you have a reasonably short 
written summary, simply attach a copy.)] 



9-0 SPECIFIC PURPOSES: 

[Name two specific examples of questions or operational problems 
this MSG has been used to answer.] 
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[NOTE: If more than one category applies, circle more than one 

answer in your reply.] 

#10. Purpose: This question is somewhat redundant with #8 and #9. 

However, here you are asked to conform to a few-word description of the 
work. It is important to note the distinctions made in #10-3 and #10-4. 
Some strategic games have introduced diplomatic considerations and 
international bargaining. These would fall under the category #10-3. 
Other simulations and gaming exercises may be explicitly concerned with 
internal economic and domestic repercussions. These would more appro- 
priately be classified under #10-4. It is of course possible to have 
exercises that may be classified under more than one category, such 
as both #10-3 and #10-4. 



* 



10-0 


PURPOSE 


10-1 


PURPOSE 


10-2 


PURPOSE 


10-3 


PURPOSE 


10-4 


PURPOSE 


10-5 


PURPOSE 


10-6 


PURPOSE 


10-7 


PURPOSE 



TECHNICAL EVALUATION 
DOCTRINAL EVALUATION 
FORCE STRUCTURE EVALUATION 

MILITARY-DIPLOMATIC ANALYSIS/INTERNATIONAL RELATIONS 
Mil ITARY -POLITICAL-ECONOMIC ANALYSIS/DOMESTIC RELATIONS 
TRAINING/EDUCATION 
RESEARCH METHODOLOGY 

OTHER ( ) Specify 



Low Middle. High 

i 1 1 

Confidence Level 



#11. Classification: Often simulations or models per se are unclas- 

sified. However, when military planning factors are added they then 
become secret or may have an even higher classification. This means that 
for some there will be supportive documents that fall into many classi- 
fications. #11-5 Proprietary. We include business classifications, 
such as simulations run internally by corporations and classified as 
corporate confidential; furthermore, proprietary implies that the 
documents are made available only by the proprietor to whomever he 



sees 


fit. 








11-0 


CLASSIFICATION 


(WITHOUT 


INPUTS) 


NA; UNCLASSIFIED 


11-1 


CLASSIFICATION 


(WITHOUT 


INPUTS) 


F0U0 


11-2 


CLASSIFICATION 


(WITHOUT 


INPUTS) 


CONFIDENTIAL 


11-3 


CLASSIFICATION 


(WITHOUT 


INPUTS) 


SECRET 


11-4 


CLASSIFICATION 


(WITHOUT 


INPUTS) 


TOP SECRET 


11-5 


CLASSIFICATION 


(WITHOUT 


INPUTS) 


PROPRIETARY 


11-6 


CLASSIFICATION 


(WITHOUT 


INPUTS) 


OTHER ( 



Specify 



O 
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12-0 CLASSIFICATION (INPUT 
12-1 CLASSIFICATION (INPUT 
12-2 CLASSIFICATION (INPUT 
12-3 CLASSIFICATION (INPUT 
12-4 CLASSIFICATION (INPUT 
12-5 CLASSIFICATION (INPUT 
12-6 CLASSIFICATION (INPUT 



DATA, OTHER INFORMATION*) 
DATA, OTHER INFORMATION*) 
DATA, OTHER INFORMATION*) 
DATA, OTHER INFORMATION*) 
DATA, OTHER INFORMATION*) 
DATA, OTHER INFORMATION*) 
DATA, OTHER INFORMATION ) 
) Specify 



NA;UN CLASSIFIED 
FOUO 

CONFIDENTIAL 
SECRET 
TOP SECRET 
PROPRIETARY 
OTHER ( 



*[DESCRIBE] 



#13. Professional Review. What professional reviewing procedures are 
used to check periodically on the validity of this MSG, its use and 
its inputs? Is there any formal reviewing group external to both the 
users and producers? 



* 

13-0 PROFESSIONAL REVIEW: 



WHO? ( 

LAST DONE? ( 



) 



#14. MSG Parent or Antecedents; The first three subcategories 
ask for information on a formal model parent, i.e., existing model, 
simulation, or game that is acknowledged as the direct parent of the 
model categorized. Even for original models, there in general was a prior 
work or specific piece of literature or several models and pieces of 
literature which provided the suggestion or stimulation (positive or 
negative) for the construction of this model. 

14-0 MSG PARENT OR ANTECEDENTS, DIRECT: NONE 

14-1 MSG PARENT OR ANTECEDENTS, DIRECT: ONE 

14-2 MSG PARENT OR ANTECEDENTS, DIRECT: MORE THAN ONE ( 

__) Specify 

14- 3 MSG MODEL PARENT OR ANTECEDENTS, INDIRECT, NUMBER: ( ) 

Low Middle High 

i 1 i I 

Confidence Level 

15- 0 NAME DIRECT PARENTS ( ) 

( ) 
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16-0 NAME INDIRECT ANTECEDENTS: ( ) 

* 

f/17. Development Initiation Date: Give date or approximate date to 

nearest month or year. 

*17-0 DEVELOPMENT INITIATION DATE: ( ) 

* 

#18 . Use Initiation Date: This means the first production run. 

18-0 USE INITIATION DATE: ( ) 



#19 . MSG Spinoff. This refers to models, games or simulations for 
which this MSG was a parent. We also count the same MSG used elsewhere 
by a different agency, authority, or group in which case "same" would 



be entered as a name of a spinoff. 




19-0 


MSG SPINOFF: 


NONE 




19-1 


MSG SPINOFF: 


ONE OR MORE ( 


) Specify number. 



Low 


Middle 


High 


i 


i i 


1 



Confidence Level 



20-0 NAMES OF ONE OR TWO SPINOFFS : ( 

( 



> 

) 




O 

A,_ 



1 
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MODEL/ SIMULATION/ GAME PRODUCTION AND PURPOSE 



# 21. Funding: In some instances , especially when individuals do a 

fair amount of work in their "spare time," one should note more than 
one funding source. Multiple funding sources may also arise when a 
game is started at one location and completed, run, or used at another 
location. NSF would be classified as #21-1 and #21—9. 



21-0 FUNDING SOURCE: ( ) 

21-1 FUNDING SOURCE: FOUNDATION ( ) 

21-2 FUNDING SOURCE: PRIVATE (BUSINESS, SELF, MISC. ) ( ) 

21-3 FUNDING SOURCE: ARPA ( ) 

21-4 FUNDING SOURCE: JCS ( ) 

21-5 FUNDING SOURCE: USA ( ) 

21-6 FUNDING SOURCE: USAF ( 6 ) 

21-7 FUNDING SOURCE: USN ( ) 

21-8 FUNDING SOURCE: OTHER DOD ( ) 

21-9 FUNDING SOURCE: OTHER US GOVERNMENT ( ) 

21-10 FUNDING SOURCE: UNIVERSITY ( ) 

21-11 FUNDING SOURCE: OTHER ( ) 






#22. MSG Production: #22-2 For profit includes 

a game is built by a for-profit organization but 
as a product. For example, it might be used for 
internal training purposes. The producer may be 



tion building for the 


DOD. 


22-0 


MSG PRODUCTION: 


PRIVATE, INDIVIDUAL 


22-1 


MSG PRODUCTION: 


UNIVERSITY 


22-2 


MSG PRODUCTION: 


FOR PROFIT 


22-3 


MSG PRODUCTION: 


ARMED FORCES 


22-4 


MSG PRODUCTION: 


NOT FOR PROFIT 



the possibility that 
not necessarily used 
research or for 
a for-profit organiza- 
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#23. MSG Initiation; #23-1 Model builders /researchers refers to a 
project where the original proposal was initiated with the individual 
specifically interested in researching and building the simulation. 

Much of MSG/research work falls under this category. The researchers 
propose the construction of the MSG to a funding or sl sponsoring agency. 
#23-2 MSG/users may initiate a proposal for construction. For example, 
a decision may be made to run a model at an institution which, has a 
special facility. The request is made to the constructors and when the 
simulation is ready, those who requested it actually participate in its 
operation. 



#23-3 There may be a request made internally to an organization from 
a management group for the construction of a model to be used by other 
individuals in the organization. In other words, "a” requests "b" to 
construct a model to be used by M c. ,f This is a fairly common managerial 
structure. #23-4 An agency outside of an organization which intends to 
use a model may request the organization to build it for the agency. 

For example, the local governmental group may decide to have several 
sessions with an urban development simulation. They may request private 
corporations to build it for them and may then use the model for training, 
operational, or advocacy purposes. #23-5 An outside agency may request 
a different institution to both build and use a simulation for their 
purposes. In this case, the sponsor is really only interested in the 
results and not in the specific aspects of the model • For example, in 
certain simulations or computerized battle models, the question posed 
may be f, what are the characteristics of this weapon under a given set 
of circumstances? 11 The sponsor may approve having a simulation built and 
constructed; however, the sponsor T s technical interest per se, is only 



NA; UNKNOWN 

MODEL BUILDERS/RESEARCHERS 
MODEL USERS INTERNAL TO ORGANIZATION 
INTERNAL TO ORGANIZATION 
USER, EXTERNAL TO ORGANIZATION 
NON-USER, EXTERNAL TO ORGANIZATION 



limited to the results 


23-0 


MSG 


INITIATION: 


23-1 


MSG 


INITIATION: 


23-2 


MSG 


INITIATION: 


23-3 


MSG 


INITIATION: 


23-4 


MSG 


INITIATION: 


23-5 


MSG 


INITIATION: 




oq 
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#24. Initiator Purpose: #24-1 The differentiation we wish to make 

between the terms teaching and training is that training is more 
concerned with "how to" whereas teaching is more concerned with "why." 

In many instances teaching and training blend imperceptibly into each 
other. #24-2 Analysis. This meaning is to be distinguished from 
#24-5 research/ theory development. Analysis means the honest grappling 
with a specific question or set of questions related to a given problem. 
The distinction is best made between operational modeling, where a purpose 
of analysis is fairly well known, and academic mo deling, where research 
and theory development are more the norm. 



24-0 


INITIATOR PURPOSE: 


NA; UNKNOWN 


24-1 


INITIATOR PURPOSE: 


TEACHING/TRAINING 


24-2 


INITIATOR PURPOSE: 


ANALYSIS/DIAGNOSTICS 


24-3 


INITIATOR PURPOSE: 


OPERATIONAL 


24-4 


INITIATOR PURPOSE: 


EXPERIMENTAL 


24-5 


INITIATOR PURPOSE: 


RESEARCH/THEORY DEVELOPMENT 


24-6 


INITIATOR PURPOSE: 


POPULARIZATION, ADVOCACY 


24-7 


INITIATOR PURPOSE: 


OTHER ( 



Low Middle High 

1 i i L 

Confidence Level 

#25. Specificity of Purpose of Funding Source: The specificity of 

purpose of the funding source will in general depend heavily upon 
whether or not the funding source was also the initiator of the game 
project. For example, a proposal may be made to ARPA to study uncon- 
ventional warfare. They will sketch out certain aspects of their 
proposal and more or less what they are going to do. The decision to 
fund this will be based upon whether or not the group appears to be 
competent and the work seems to be "reasonable," and a considerable 
amount of leeway will be left for the group's actual work- 
Such a si tuation would fall under il 25—2 moderately specified or #25-3 
diffusely specified. On the other hand, there may be a specific request 
from a government agency to test a specific piece of equipment and to 
use a study or simulation to report on the quality of this equipment. 



25-0 SPECIFICITY OF PURPOSE OF FUNDING SOURCE: 
25-1 SPECIFICITY OF PURPOSE OF FUNDING SOURCE: 
25-2 SPECIFICITY OF PURPOSE OF FUNDING SOURCE: 
25-3 SPECIFICITY OF PURPOSE OF FUNDING SOURCE: 



NA; UNKNOWN 
TIGHTLY SPECIFIED 
MODERATELY SPECIFIED 
DIFFUSELY SPECIFIED 



Low Middle High 



Q 

i > _ 

Confidence Level 



1 



1 
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#26. Best Alternative Procedures? Imagine that the objectives of the 
simulation must be achieved by a different means. It might appear that 
to check none and then to check something else would be mutually in- 
consistent. However, if the MSG is used for more than one purpose this 
could easily arise. For example, in a MSG used for experimentation 
there may be no alternative for the experiment, yet the model may also 
be used for teaching. In this case, lectures or case studies would be 
reasonable alternatives. 



26-0 


BEST 


ALTERNATIVE 


PROCEDURES? 


26-1 


BEST 


ALTERNATIVE 


PROCEDURES? 


26-2 


BEST 


ALTERNATIVE 


PROCEDURES? 


26-3 


BEST 


ALTERNATIVE 


PROCEDURES? 


26 -A 


BEST 


ALTERNATIVE 


PROCEDURES? 


26-5 


BEST 


ALTERNATIVE 


PROCEDURES? 


26-6 


BEST 


ALTERNATIVE 


PROCEDURES? 



N A; UNKNOWN 

NONE OR VIRTUALLY NONE 
LECTURES 

CASE STUDIES /HISTORY 

ANALYSIS 

EXPERIENCE 

GAMING 



#2 7. Major Use of MSG: Select the appropriate categories from the 

list below. Also provide written commentary to explain what you mean 
by the categoriee selected, i.e., give for instances. 



27-0 MAJOR 
27-1 MAJOR 
27-2 MAJOR 
27-3 MAJOR 
27— A. MAJOR 
27-5 MAJOR 
27-6 MAJOR 
27-7 MAJOR 



USE OF MSG 
USE OF MSG 
USE OF MSG 
USE OF MSG 
USE OF MSG 
USE OF MSG 
USE OF MSG 
USE OF MSG 



NA; UNKNOWN 

TEACHING/TRAINING 

ANALYSIS/DIAGNOSTICS 

OPERATIONAL 

EXPERIMENTAL 

RESEARCH/THEORY DEVELOPMENT 
POPULARIZATION , ADVOCACY 
OTHER ( ) 



Low Middle High 

i i i 

Confidence Level 



#28. Analysis Procedures; Explain, providing for instances , the type, 
amount, and rigor of analytic procedures used on the output of this MSG 




Low Middle High 

I i l l 

Confidence Level 
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It 29. Judged Effectiveness of Best Alternative Procedure: This is 

for the main purpose of the MSG. By main purpose of the MSG we mean 
the main use that in fact has been made of it. Initiator purpose had 
the possibility of alternative procedures for more than one use. In 
answering this question we restrict ourselves only to the major use. 

29-0 JUDGED EFFECTIVENESS, BEST ALTERNATIVE PROCEDURE: ( 

) 

Low Middle High 

I ! 1 I 

Confidence Level 



it 30. 
* 


Number 


of 


Briefings : 


Total number 


to date based on this MSG. 


30-0 


NUMBER 


OF 


BRIEFINGS : 


UNKNOWN 




30-1 


NUMBER 


OF 


BRIEFINGS : 


( 


) Specify 



#31. Level of Briefing: By level of briefing we refer to organization 

and organizational level and a description of the personnel level involved. 
For example, number and rank of generals or senior government officials 
present. 

* 

31-0 LEVEL OF BRIEFING: [Describe] 



It 32. Purpose of Briefing: Identify and state purpose of two most 

important briefings, i.e., what decisions depended on this MSG? 



32-0 






PURPOSE OF BRIEFING: 



[Describe] 
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#33. Importance of MSG to Decision: What impact did MSG have on 

decision just noted? Describe specifically. 



* 

# 34. Measure of Benefits: Specify what you regard as a reasonable 

measure of benefits and success from this >iSG. 



MSG PRODUCTION COSTS 



#35. Direct Costs to Build: Under this category we mean actual 

expenditures of money earmarked for the purposes of developing this 
particular game, formal pay for working time, and measures attributed 
to the cost of the game. In many instances where there is a great amount 
of informal work done, the direct cost for the development of the game 
may be zero, although the indirect and unallocated costs may be enormous. 
For example, the UCLA business game was developed heavily by the use of 
faculty time which was not particularly assigned to game building. Many 
university games have this property. Many games built in-house to an 
institution without direct contractual assistance also have this property. 
Thus, the question should be construed as one of finding out when direct 
monies were assigned for the purpose of game construction. 



35-0 DIRECT COSTS TO BUILD: 
nearest $10,000. 



( ) Specify to 

Low Middle High 

! I — 1 

Confidence Level 
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#36. The question concerning funding for development and building is 
specifically directed at the accounting question of what money has been 
formally assigned to the effort concerning a specific model, simulation 
or game. Thus it will almost always be an underestimation of cost. 

36-0 DIRECT FUNDING TO BUILD: NO 
36-1 DIRECT FUNDING TO BUILD: YES 



37-0 AMOUNT OF FUNDING TO BUILD: ( ) Specify 

Low Middle High 

' '} 

I 1 1 1 

Confidence Level 



#38. Total Costs Direct, Indirect, Imputed and Unimputed: The 

ranges are purposely kept relatively wide open because of the extreme 
difficulties in accurately judging the costs. In some cases a variance 
of 100% or more is to be expected. This will probably be a rather 
surprisingly large number if one is honest with himself. There is an 
obvious political problem here. Many of the unimputed costs would in the 
course of time have an opportunity cost of zero. Furthermore, to a great 
extent many of the unimputed costs are extra hours of work put in by 
oneself and not paid for. Hence they do not necessarily come out of any- 
body’s budget or funding. They might be called the "Out of Hide Costs." 

* 

38-0 TOTAL COSTS TO BUILD, DIRECT, INDIRECT, IMPUTED, 

UNIMPUTED: ( ) Specify to nearest 

$10 ’ 000 - Low Middle High 

I I 1 i 

Confidence Level 




28 
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MODEL/SIMULATION/GAME OPERATING COSTS AND OPERATIONS 



#39. Under costs to operate we include professional time, support 
time, set-up time, computer time, and experimental subject of participant 
time together with overhead. In many cases many of these items are pro- 
vided free. As we are discussing actual expenditures, the estimates we 
will be asking for will be gross underestimates. 

There is possibly more variability in operating coats than in build- 
ing costs for some types of games. The variability comes in the way 
player time is counted as a cost as well as facilities' use and operator's 
time. For example, with many business games played at universities it 
would appear that the financial costs are zero, as the professorial and 
student time is not regarded as a direct cost and in some instances 
computation does not appear as a directly imputed cost and the use of 
facilities which would otherwise be unoccupied is deemed to be free. 

Any attempt to replicate that game in an environment that does not 
have these features may be extremely costly. 

A political military exercise is usually run as an individual affair, 
and we should consider further replications even though they might be 
regarded as part of the same experiment as separate items. With educa- 
tional games or time-shared experiments, costs are calculated based on 
individual game or subject use. For simulation the cost to operate is 
the cost to explore the answer to a typical question. This is relatively 
vague, but an upper bound would be the amount of exploration that might 
result in a small separate publication. 



39-0 ANNUAL COST TO OPERATE, GROSS: 

[An adequate answer to the question may require written comment. 

If so, please write below.] 

Low Middle High 



YEAR COST 



19 


( 


) 


i i — i — 

Confidence Level 


19 


( 


) 




19 


( 


) 


Specify for last five years. 


19 


( 


) 


19 




) 





29 
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#40. Cost to Operate, Single Use: What does it cost for a single use 

of the MSG, assuming only minor or no variations in input values? For 
example, name such a use and date it approximately. By use, we mean 
for a single study effort. (This of course may vary, but give an 
average estimate.) 

* 

40-0 COST TO OPERATE, SINGLE USE: 



#41. Annual Update Costs: If the model is under continuous development, 

what are the annual’ costs of these activities, over and above "normal 11 
operating costs? 



* 



41-0 .ANNUAL UPDATE 
41-1 'ANNUAL UPDATE 
41-2 ANNUAL UPDATE 



COSTS, TOTAL DOLLARS: ( 

COSTS, PROFESSIONAL MAN-YEARS: 
COSTS, PROGRAMMER MAN-YEARS : ( 



) 

> 



Low Middle High 

I I I I 

Confidence Level 



#42. Operational Life Span (to date): By operational life span we mean 

the period starting after development is complete, beginning with the first 
operational use of the model to the period when it and its analysis or 
post-run exposition are set aside, having served their purpose. 

If a model, simulation, or game that has been in more or less continuous 
use is still in operation, both that length of time of operation and the 
fact that it is still operational should be indicated. 

* 

42-0 OPERATIONAL LIFE SPAN (TO DATE) : ( ) Specify 



Low Middle High 




L 



Confidence Level 



a 
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//43. Still Active?: By this we mean, "Is the MSG in its original or 

mildly modified form still actively being used for production runs?" 
We contrast this with a serious revision that has resulted in either 
a new name or an appellation such as Mark II, or Mod III, and so on. 

* 

43-0 STILL ACTIVE?: NO 

43-1 STILL ACTIVE?: YES 

Low Middle High 



Confidence Level 



* 

44-0 MODEL USERS: 

[Specify all agencies who directly use the MSG outputs by full 
name (not military abbreviations); indicate the prime user.] 



* 

// 45. Operational Use Annual Frequency: By this we are trying to find 

out how many times this MSG is actually exercised or run annually. 

YEAR NUMBER OF TIMES 



Specify for 
last five years 



46-0 EXPERIMENTAL USE: NO 

46-1 EXPERIMENTAL USE: YES 



#47. Experimental Example: If 46-1, i.e. , "yes," give a for instance. 

[Describe] 



19 

19 

19 

19 

19 
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#48. Experimental Purpose, Initial: Was this MSG designed originally 

for experimental use? 



48-0 EXPERIMENTAL PURPOSE, INITIAL: NO 
48-1 EXPERIMENTAL PURPOSE, INITIAL: YES 



49-0 EDUCATIONAL USE: NO 

49-1 EDUCATIONAL USE: YES 



#50. Educational Example: If 49-1, i.e., "yes," give a for instance. 

[Describe] 



#51. Educational Purpose, Initial: 
for educational use? 

51-0 EDUCATIONAL PURPOSE, INITIAL: 
51-1 EDUCATIONAL PURPOSE, INITIAL: 



Was this MSG designed originally 

NO 

YES 




OO 
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#52. Transferability of MSG Use: #52-0 is a model not intended for 

transfer. An example of such might be a classified simulation run to 
test for some particular parameter value and dispensed with after the 
runs; or a study or simulation may be extremely classified with only 
one user in mind. #52-1 This would be something like chess or Go where 
once the rules have been transmitted, one could take a piece of paper, 
draw the board, obtain some stones, and play. In general, a game of 
this variety can be transferred immediately at little or no cost. #52-2 
An example of a game in this category would be a game that is not overly 
complex, is extremely well documented and produced, and is generally 
available. The games Simsoc or Whifn T proof or Summit or Democracy would 
all fit into this category. 

52-3 Middling would cover simulation games such as the UCLA business 
game where the size of the program is not enormous and computer require- 
ments are such that, at least in the United States, many institutions 
would have the facilities. Furthermore, the documentation is reasonably 
good. To get it operating, such a game would require one or two months, 
taking into account telephone calls, time delays in the mails, etc. 

52-5 Simulations that are extremely difficult to travel are ones that 
depend upon specific facilities and crews of experienced individuals or 
that are enormous in computational size. For that matter they may 
not be computer games, but have become so large that they should be 
regarded more as institutions rather than games. Examples of such are 
METRO, The RAND Logistic Lab experiments, TEMPER, and the Carnegie Tech 
game. The reasons why these cannot travel are different; however, in 
each case the amount of work required to transfer the operation 
is enormous. 



* 



52-0 


TRANSFERABILITY 


OF 


MSG 


USE: 


52-1 


TRANSFERABILITY 


OF 


MSG 


USE: 


52-2 


TRANSFERABILITY 


OF 


MSG 


USE: 


52-3 


TRANSFERABILITY 


OF 


MSG 


USE: 


52-4 


TRANSFERABILITY 


OF 


MSG 


USE: 


52-5 


TRANSFERABILITY 


OF 


MSG 


USE: 



NOT INTENDED FOR TRANSFER 
GENERALLY 

MODERATE DIFFICULTY 
MIDDLING DIFFICULTY 
DIFFICULT 

EXTREME DIFFICULTY 



[Describe whether transferability just indicated pertains to any of 
the following situations: (1) use by another person or organization at 

a new site, (2) use by the same developer/designer/builder at a new site, 
or (3) use by another person or organization at site where MSG presently 
operates. ] 




'h 
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// 53. Transferability Costs to Operate: Using information provided 

in //52, please estimate how much additional amount, with respect to 
normal operating costs, would be required to transfer and then operate 



this 


MSG. 
















53-0 


TRANSFERABILITY 


COST 


TO 


OPERATE : 


NA; UNKNOWN 








53-1 


TRANSFERABILITY 


COST 


TO 


OPERATE : 


COST 








53-2 


TRANSFERABILITY 


COST 


TO 


OPERATE : 


COST TO (COST + 


10%) 






53-3 


TRANSFERABILITY 


COST 


TO 


OPERATE : 


(COST + 10%) TO 


(COST 


+ 


25%) 


53-4 


TRANSFERABILITY 


COST 


TO 


OPERATE: 


(COST + 25%) TO 


(COST 


+ 


50%) 


53-5 


TRANSFERABILITY 


COST 


TO 


OPERATE : 


(COST + 50%) TO 


(COST 


+ 


100%) 


53-6 


TRANSFERABILITY 


COST 


TO 


OPERATE : 


> (COST + 100%) 









Low Middle High 

I 1 i 

Confidence Level 



#54. Obsolescence: Comment on the speed at which you expect this 

MSG to become obsolescent indicating the reasons why. We are not 
referring to the need for reprogramming for new hardware or for minor 
modifications, but to the state where it is no longer sufficiently 
relevant that either a major modification has to be made or it is 
completely abandoned. For example, a special simulation may be built 
and run once for a specific purpose. A simple model may be used for 
many years, as long as the type of damage calculation it performs 
is relevant. 




34 
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#55. Related MSG: Does there exist an MSG that might be regarded as 

serving approximately the same purpose as yours? 

55-0 RELATED MSG: NO 
55-1 RELATED MSG: YES 



#56. Duplication of Use: If 55-1, i.e. "yes," name the MSG(S) 

and state why or why not one MSG could serve the purposes of all. 
If your answer is "no," i.e., 55-0, state why you believe there is 
no MSG similar to yours. 



#57. Clearing House: Is there enough communication in DOD among 

different gaming and simulation studies or would a clearing house 
or central agency be of use? 



* 



57-0 


CLEARING HOUSE: 


HIGHLY 


USEFUL 


57-1 


CLEARING HOUSE: 


USEFUL 




5 7-2 


CLEARING HOUSE: 


SAME AS 


IS 


57-3 


CLEARING HOUSE: 


HARMFUL 




57-4 


CLEARING HOUSE : 


HIGHLY 


HARMFUL 



Comments on Question #57 are required: 
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#58. Standardization: After techniques and studies have been in 

existence for some time, standardization and exchange of common 
routines is extremely useful. However, some times premature attempts 
to standardize do more harm than good. In particular, redundancy to 
an outsider may' not be redundancy to those doing a study. What is 
your belief in the advisability of increasing DOD gaming and simulation 
activity for standardization? 

* 



58-0 STANDARDIZATION: 
58-1 STANDARDIZATION: 
58-2 STANDARDIZATION: 
58-3 STANDARDIZATION: 
58-4 STANDARDIZATION: 



HIGHLY USEFUL 
USEFUL 
SAME AS IS 
HARMFUL 

HIGHLY HARMFUL 



Comments on Question #58 are required: 



Low Middle High 

1 1 1 1 

Confidence Level 



#59. Regional Centers: Models, simulations, and games are operated 

and built at many locations using many languages and different staffs 
and equipment. Would, for example, 3-5 appropriately cleared regional 
centers for most of these activities be a more or less effective way 
of supporting them? 

Discuss: 




36 



31 



0 



#60. External Review Board: Would you comment on reasons both pro 

and con having an external review board consider this and other MSG's 
built/operated/used by your organization or activity? 

Discuss: 



#61. External Review Board, Specification: Would you characterize 

a "good" external review board as to composition and function? 

Discuss : 



n*y 

t 



o 
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PART II 



MODEL/SIMULATION/GAME CHARACTERIZATION & DESCRIPTION 



NOTE: 

If Part II not filled out by Office of Prime Responsibility and/or 
user, please indicate who filled out Part XI. 



Name 

Organization 

Relationship to 
Prime Office of 
Responsib ility_ 



3 
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#62. Scenario Type:. #62— 0 means that the inputs, outputs and inter- 
pretation of the outputs are all numerical (example: number of targets 

destroyed). #62-1 means that a verbal description of the scenario must 
be available prior to use (example: a man— machine or free-form game) . 

# 62-2 means that the interpretation of an output is qualitatively 
modified or interpreted prior to being used (example: a written 

assessment of qualitative aspects of target damage may accompany a 
computer output). 



62-0 SCENARIO TYPE: ONLY NUMERICAL 

62-1 SCENARIO TYPE: VERBAL DESCRIPTION NEEDED FOR USE 
62-2 SCENARIO TYPE: VERBAL DESCRIPTION NEEDED FOR ANALYSIS 



#63. Scenario Description: Rich "realistic" may be used to refer to 

a scenario which is both rich in detail and purports to be a realistic 
description of some phenomenon. For example, some tactical games may 
go to great lengths to have a realistic description of weather conditions, 
troop morale conditions, terrain conditions, details on buildings, and 
so forth. The measure of the description of environment should be 
relative to the real world phenomenon being modeled. For example, a 
business game might have fewer details in it than a diplomatic game 
but be a richer model in relation to actuality than the diplomatic 
game. Furthermore, some games may have underneath them a mathematical 
model which is not necessarily apparent to the users. The word "imaginative" 
can be used to refer to nonfactual modeling where a scenario may contain 
counterfactual or futuristic features. These aspects may be mixed in 
with other environmental categorizations. 

* 

63-0 SCENARIO DESCRIPTION: [Describe] 




39 
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if 64. Mathematical Sophistication of MSG: #64—0 , None equals less 

than high school maths needed to interpret output or participate as 
a player if it is a game. #64—1 , Slight equals high school maths. 
#64 — 2 , Moderate equals needs college level maths (BA) or engineering 
degree. #64-3, High equals requires an advanced degree to interpret 
the output. 



64-0 


MATHEMATICAL 


SOPHISTICATION 


OF MSG: 


NONE 


64-1 


MATHEMATICAL 


SOPHISTICATION 


OF MSG: 


SLIGHT 


64-2 


MATHEMATICAL 


SOPHISTICATION 


OF MSG: 


MODERATE 


64-3 


MATHEMATICAL 


SOPHISTICATION 


OF MSG: 


HIGH 










Low Middle 



I I i 1 

Confidence Level 



#65. Timing of Moves: Event timing implies that moves depend upon 

a specific event having occurred. Fixed clock timing implies that there 
is a certain increment of time upon which model activities are based. 
There are some models that are both event and fixed clock, in the sense 
that generally the clock moves forward at regular periods, e.g., descrip- 
tions of gross national product in an international model. However, 
simultaneously there may be moves that depend on specific events, such 
as conditional checking for threshold effects, time in queues, etc. 



65-0 

65-1 

65-2 



TIMING OF MOVES : 
TIMING OF MOVES : 
TIMING OF MOVES : 



NA; UNKNOWN 
EVENT 

FIXED CLOCK 



#66. Model Time to Real Time Ratio: In describing the ratio of model 

time to real time one has the problem of distinguishing between the 
period assigned the real time and the amount of that time which would 
have been used for the decisionmaking. For example, in a model in which 
the real time is meant to be quarters, the price decision in a market may 
only take a week or two to make. In the exercise, twenty minutes may be 
allotted for the decisionmaking. We now have the problem of deciding 
whether to scale the twenty minutes against the one week or the three 
months. We suggest scaling against the allotted real time, i.e., the three 
months. 

66-0 MODEL TIME TO REAL TIME RATIO: ( ) Specify 

Low Middle High 




i i — 

Confidence Level 



40 



L 



j 
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#67. Time Represented: In some instances where the simulator is 

not necessarily meant to represent any specific structure the time 
period represented might be interpreted as the present, but it might also 
be better to describe it as unspecified. In cases of doubt, it is best 
to note the model in both categories . 



6 7-0 


TIME 


REPRESENTED: 


PAST 


6 7-1 


TIME 


REPRESENTED: 


PRESENT 


6 7-2 


TIME 


REPRESENTED: 


FUTURE 


67-3 


TIME 


REPRESENTED: 


UNSPECIFIED 


6 7-4 


TIME 


REPRESENTED: 


NOT RELEVANT 



#68. Level of Resolution, Model Time: This is the smallest time unit 

recognized by the game. 



6 8-0 


LEVEL 


OF 


RESOLUTION, 


MODEL TIME : 


NA; UNKNOWN 


68-1 


LEVEL 


OF 


RESOLUTION, 


MODEL TIME : 


SECONDS 


68-2 


LEVEL 


OF 


RESOLUTION, 


MODEL TIME : 


MINUTES 


6 8-3 


LEVEL 


OF 


RESOLUTION, 


MODEL TIME: 


HOURS 


68-4 


LEVEL 


OF 


RESOLUTION, 


MODEL TIME: 


DAYS 


6 8-5 


LEVEL 


OF 


RESOLUTION, 


MODEL TIME: 


WEEKS 


6 8-6 


LEVEL 


OF 


RESOLUTION, 


MODEL TIME: 


QUARTERS 


68-7 


LEVEL 


OF 


RESOLUTION, 


MODEL TIME : 


YEARS 


68-8 


LEVEL 


OF 


RESOLUTION, 


MODEL TIME: 


>YEARS 



Low Middle . High 

i i j j 

Confidence Level 



#69. Level of Resolution, Space: In military games, the spatial level 

of resolution is frequently important; inmost business games spatial 
level of resolution is at best crude. //69-4 refers to the situation where 
detail may be supplied for specific locations, but no detail is given 
between them: for example, details of the terrain around enemy airports, 

but no details for terrain between them. 



69-0 


LEVEL 


OF RESOLUTION, 


SPACE: 


NA 


69-1 


LEVEL 


OF RESOLUTION, 


SPACE: 


SMALL AREAS (METERS) 


69-2 


LEVEL 


OF RESOLUTION, 


SPACE: 


MODERATE (KILOMETERS) 


69-3 


LEVEL 


OF RESOLUTION, 


SPACE: 


LARGE AREA (THEATER/ CONTINENT) 


69-4 


LEVEL 


OF RESOLUTION, 


SPACE : 


VARIED 



Low Middle High 



\ 1 \ i 

Confidence Level 




0 
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#70. Level of Resolution, Sides: In some models for some purposes there 

is no need to resolve the nature of individual teams. Gross performances 
of the interaction as a whole are being considered regardless of team size. 
For other purposes the same model may be used with considerable attention 
paid to the team structure. 

Furthermore, a distinction between structured and unstructured groups 
must be made. In some instances, e.g., when studies of the emergence of 
leadership are being conducted, it is important that no structure be 
placed on the teams. In other cases the teams may be given a structure 
such as that of a corporation or a military command. 

#70-7. As platoon, division, air force, etc., vary in size between the 
services, name the generic term for the unit. 



70-0 LEVEL OF RESOLUTION, SIDES: NA 

70-1 LEVEL OF RESOLUTION, SIDES: INDIVIDUALS 

70-2 LEVEL OF RESOLUTION, SIDES: SMALL GROUPS ( STRUCTURED) 

70-3 LEVEL OF RESOLUTION, SIDES: ORGANIZATIONS, SMALL (FEW-100) 

70-4 LEVEL OF RESOLUTION, SIDES: ORGANIZATIONS, LARGE (1000'S) 

70-5 LEVEL OF RESOLUTION, SIDES: ORGANIZATIONS, VERY LARGE 

70-6 LEVEL OF RESOLUTION, SIDES: SMALL GROUPS, UNSTRUCTURED 

70-7 LEVEL OF RESOLUTION, SIDES: NAME UNIT ( ) 



Comment : 



Low 



Middle High 



Confidence Level 



ERJC 
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it 71. Level of Resolution, Military Action: The categories here are 

arranged in order of progressive generality, thus it 71-5 includes war as 
a part of ongoing diplomacy. 



71-0 LEVEL OF RESOLUTION, MILITARY ACTION: 
71-1 LEVEL OF RESOLUTION, MILITARY ACTION: 
71-2 LEVEL OF RESOLUTION, MILITARY ACTION: 
71-3 LEVEL OF RESOLUTION, MILITARY ACTION: 
71-4 LEVEL OF RESOLUTION, MILITARY ACTION: 
71-5 LEVEL OF RESOLUTION, MILITARY ACTION: 



NA; UNKNOWN 

ENGAGEMENT 

BATTLE 

CAMPAIGN 

WAR 

DIPLOMATIC 



Comment: [Note specific details of this MSG.] 



Low Middle High 

I i 1 l 

Confidence Level 



it! 2. Random Events: It is possible to use a model occasionally with 

random events and on other occasions without random events. In this 
case both categories should be checked. One should not regard this 
question as a binary choice; both are possible. 

72-0 RANDOM EVENTS: NO 

72-1 RANDOM EVENTS: YES 

Comment: [Note specific details of this MSG.] 
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CHARACTERIZATION 
PLANNING FACTORS AND DATA 



#73. Data Sources and Validity: Where did the data for this MSG come 

from? List sources in as complete a fashion as possible. Were any 
independent checks performed to insure the accuracy, timeliness, 
consistency, and overall quality of the data? Describe them. 

* 

73-0 DATA SOURCES AND VALIDITY: [Comments] 



#74. Types of Data: It has been suggested that three types of data can 

be distinguished in games and simulations: 

Type 1 data = High certainty data 

Examples: range of a weapon under specified conditions, the 

size of a unit of troops, etc. 

Type 2 data = Certain level of uncertainty 

Examples: outcome of a company fight, radar detection range 

(these need parametric studies and sensitivity 
analysis for validation) . 

Type 3 data = High uncertain and hard to test 

Examples: diplomatic behavior, enemy goals, broad social 

or economic reactions to policy. 



* 

74-0 TYPES OF DATA 

[In terms of the three types of data requirements describe the data 
needed for your game or simulation.] 




,'h 
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//75. Number of Inputs (Constants, Parameters, and Variables) in MSG: 

This may vary from use to use; therefore, if necessary, give lower bound, 
average, and upper bound. 



I 75-0 NUMBER OF INPUT CONSTANTS: 

75-1 NUMBER OF INPUT PARAMETERS: 
75-2 NUMBER OF INPUT VARIABLES: 



Comment : 



( ) Specify 

( ) Specify 

( ) Specify 

Low Middle High 

I J l 1 

Confidence Level 



# 76 . Number of Output Variables in MSG: This may vary from use to 

use; therefore, if necessary, give lower bound, average, and upper bound. 



76-0 NUMBER OF OUTPUT VARIABLES: ( ) 

Specify 

• Low Middle High 

1 I I A 

Confidence Level 



Comment: 
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#77 . Intangibles: Are sometimes ruled out by limiting the scope of 

the study; b.y obtaining rulings from higher authority as to how they 
are to be treated; by using expert estimates; by using high and low 
bounding procedures or by other methods. 



77-0 INTANGIBLES 

[Describe how intangibles are treated in this MSG. In answering 
give a for instance.] 



#78. Sensitivity Analysis: Discuss the importance of sensitivity 

analysis for this MSG and describe how it is done, if it is done at all. 
We are not interested in sensitivity analysis done as a routine matter 
of debugging; rather, what has been done since the MSG has been 
operational? 

78-0 SENSITIVITY ANALYSIS: 



Comment : 



* 

79-0 DATA COLLECTION TIME: ( 



) 



[Estimate (in man-years) data collection time required.] 



Low 



Middle High 



Confidence Level 



ERIC 
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#80. Data Validation: Frequently all data come from another agency 

or source with no checks from the user groups . Sometimes a user group 
obtains its information first hand by measurement, observation, field 
tests, etc. Describe how you get your data inputs and what independent 
checks or procedures you perform to challenge the validity of the data. 



* 

80-0 DATA VALIDATION: 



MODEL OPERATION 
SUPPORTING FACILITIES 



#81. Computer Used for Running a Simulation: We mean the different 

computers for which this model has been run. In some cases there may be 
only one, and in other cases many modifications may have been issued 
for different machines. List not more than the three most frequently 
used operating systems. 

81-0 COMPUTER USED FOR RUNNING: NA 

81-1 COMPUTER(S) USED FOR RUNNING: THREE OR LESS ( 

) Specify 

Low Middle High 

l i i i 

Confidence Level 




#82. Program Language: This calls for the language in which the 

simulation has been programmed. Frequently there may be a series of 
languages. All should be noted. 

82-0 PROGRAM LANGUAGE : NA 

82-1 PROGRAM LANGUAGE (S) : ( ) Specify 

Low Middle High 

i i i i 

Confidence Level 
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#83 . Program Size: Approximately how many instructions are there in the 

language(s) noted above? 

83-0 PROGRAM SIZE: ( ) Specify 

83-1 PROGRAM SIZE: ( ) Specify 



II 84. Facilities: II 84-4 This refers to the situation where a special 

system set of languages or program may have been written to accompany 
the running and general handling of a specific model. For example, some 
models depend upon the availability of much of the specialized extra 
hardware and software. Although it is possible that the models them- 
selves can travel, much of their power is lost when the accompanying 
personnel and equipment are not available. 

84-0 FACILITIES: NA; UNKNOWN 

84-1 FACILITIES: SPECIAL BUILDING OR INSTITUTION 

84-2 FACILITIES: DEDICATED COMPUTER (UNCLASSIFIED) 

84-3 FACILITIES: DEDICATED COMPUTER/ CLASSIFIED TAPES 

84-4 FACILITIES: SPECIAL LANGUAGE, LIBRARY OR COMPUTATIONAL SYSTEM 

Low Middle High 

I 1 1 1 

Confidence Level 




48 



44 



SIMULATION /STUDY /MODEL PRODUCTION 



COSTS 



General caveat on building costs. The possibilities for obtaining 
close cost estimates for many MSG’s are difficult in the extreme. This 
is not merely a problem of gathering information; it is a problem of 
correct conceptualization of the costs that should be included to certain 
forms of work. In particular, joint costs play an enormously important 
role, thus it is not an easy matter to impute costs items such as computer 
time, use of joint educational facilities, and so forth to the costs. 

In this coding scheme we wish to stress that the costing figures presented 
should be used with extreme caution. 



#85. Development Time is a concept about which it is difficult to 
be both precise and accurate. For our purposes, we must emphasize that 
the categories indicated are crude in the extreme. We are trying to 
indicate the elapsed time between the decision to build a particular 
model and the first production run of that model. In many cases after 
a model has been used once, development goes on for many years. Thus 
our criterion may be regarded as presenting a gross underestimation of 
development time. Furthermore, additional complications appear on occasion 
as a model develops and it may change its name. A further clarification 
of this idea, according to our meaning, is the time from the inception of 
the work on construction until the first production run . . . this is 
contrasted with a debugging run: they are not the same. We specifically 

do not consider further modification after the first production run has 
taken place, even though ex post facto , the first production run is then 
regarded as "experimental." 

* 

85-0 DEVELOPMENT TIME: ( ) Specify in years or 

months . 

Low Middle High 

i l i 1 

Confidence Level 





r 



T 



T 



1 



v 
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#86. We contrast total man-years with professional man-years. Under 
category #86 we include graduate students, secretarial help, program- 
ming assistance and any other forms of voluntary contribution of 
time. These are direct man-years and do not include allocation of 
institutional administrative overhead. 

* 

86-0 DEVELOPMENT TIME: TOTAL MAN-YEARS: ( ) Specify 

Low Middle High 

i i i i 

Confidence Level 



#87. Professional man-years used in the development of a model. Under 
this category we include both professional designers and consultants. 

In many cases there are also graduate students, additional helpers, 
ordinary programming assistance, as well as an enormous amount of office 
staff. 

87-0 DEVELOPMENT TIME: PROFESSIONAL MAN-YEARS: ( ) 

Specify 



#88. Development Team Professional Profile: Describe the professional 

makeup of the development team (including consultants) . 

* 

88-0 DEVELOPMENT TEAM PROFESSIONAL PROFILE : 

Describe: Low Middle High 

I t i i 

Confidence Level 



89-0 DEVELOPMENT TIME: PROGRAMMER MAN-YEARS: 



( 



) 



Specify 
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MODEL CHARACTERIZATION 
DOCUMENTATION 



#90. Documentation: #90—1 Excellent means that the documentation is 

sufficiently good that it can be picked up elsewhere by a different 
group of people and operated with none or a minimum of long distance 
telephone calls and conferences. #90 — 3 Average means that the 
documentation exists in some form but it is moderately hard to operate 
without at least some discussions with the originators of the document. 
#90-6 Uneven/highly variable is put in to characterize simulation in 
which there is spotty documentation often indicating an evolution of 
different programmers and different groups working on the model. To 
get decent documentation one may have to search among the disarray of 
documents that are presented. 



* 

90-0 


DOCUMENTATION , 


EXTENT : 


NA; ZERO; UNKNOWN 


90-1 


DOCUMENTATION , 


EXTENT : 


EXCELLENT 


90-2 


DOCUMENTATION, 


EXTENT : 


VERY GOOD 


90-3 


DOCUMENTATION , 


EXTENT: 


AVERAGE 


90-4 


DOCUMENTATION, 


EXTENT : 


WEAK 


90-5 


DOCUMENTATION , 


EXTENT : 


POOR 


90-6 


DOCUMENTATION , 


EXTENT : 


UNEVEN /HIGHLY VARIABLE 


90-7 


DOCUMENTATION , 


EXTENT : 


UNAVAILABLE 




#91. Documentation Availability/Location: 91—5. Proprietary (classified)/ 

write author. What we mean here is that the information on the simulation 
is classified in the sense of top secret, secret, and so on. To obtain 
this information, it is necessary to write the author. This relieves the 
burden of identifying the document and approving of its transmittal to 
the author and to the people who are searching for the document. Problems 
of clearance, need to know, and so on can then be resolved between the two 
interested parties. 



91-0 

91-1 

91-2 

91-3 

91-4 

91-5 

91-6 

91-7 

91-8 



DOCUMENTATION , 
DOCUMENT AH ON , 
DOCUMENTATION , 
DOCUMENTATION , 
DOCUMENTATION, 
DOCUMENTATION , 



AVAILABILITY /LOCATION : 
AVAILABILITY/LOCATION: 
AVAILABILITY/LOCATION : 
AVAILABILITY/LOCATION : 
AVAILABILITY/LOCATION: 
AVAILAB IL ITY / LO CAT I ON : 



DOCUMENTATION , AVAILABILITY/LOCATION : 

DOCUMENTATION, AVAILABILITY/LOCATION: 
DOCUMENTATION, AVAILABILITY/LOCATION: 



N A; UNKNOWN 

OUT OF PRINT /UNKNOWN 
PROPRIETARY/NOT FOR PROFITS 
P ROPRIETARY / COMMERCIAL 
PROPRIETARY/WRITE AUTHOR 
PROPRIETARY (CLASSIFIED)/ 
WRITE AUTHOR 

PUBLIC/DEFENSE DOCUMENTATION 
CENTER 

PUBLIC/LIBRARY OF CONGRESS 
PUBLIC/PROFESSIONAL JOURNALS, 
BOOKS 



51 - 
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* 

92-0 DOCUMENTATION: GENERAL OVERVIEW 

[Describe the documentation in your own words in the remaining 
space. ] 



#93. Publication Type: #93—3. Reports /analyses, etc. This refers to 
publications, possibly generated after a series of runs, to be used as 
an official document, as a report to a higher authority, or possibly as 
even a supporting argument for a request for funds. This is in distinc- 
tion to a document which is a book or article for nonspecific purposes. 



93-0 


PUBLICATION 


TYPE: 


93-1 


PUBLICATION 


TYPE: 


93-2 


PUBLICATION 


TYPE: 


93-3 


PUBLICATION 


TYPE: 



BOOKS OR ARTICLES 
USER MANUALS 
PROGRAM DECKS /LISTINGS 
REPORTS/ANALYSES, ETC. 




CO 



p 



p 



p 



1 
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* 

94-0 PUBLICATION OR DOCUMENT IDENTIFICATION: 

[Specify one or two documents most relevant to this game simulation 
or study. Give full references so that documentation may be assembled.] 



TECHNICAL COORDINATION & STANDARDS 

#95. Technical Coordination: One might have a central clearing house 

which performs a clerical operation with no professional or evaluative 
role. One might otherwise have a staff of several permanent professionals 
whose task is to compose and to technically describe the inventory of 
models, simulation, or games. In your opinion, supposing that a central 
clearing house exists should it have a technical staff? 



95-0 


TECHNICAL 


COORDINATION 


95-1 


TECHNICAL 


COORDINATION 


95-2 


TECHNICAL 


COORDINATION 


95-3 


TECHNICAL 


COORDINATION 


95-4 


TECHNICAL 


COORDINATION 



HIGHLY DESIRABLE 
UNDESIRABLE 
INDIFFERENT 
DESIRABLE 
HIGHLY DESIRABLE 





Dis cuss : 
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7/96. Standards Committee: Question 58 asked about the desirability 

of standardization without specifying what. Is it premature to try 
to form a professional standards committee for models, games, and 
simulations. Is it needed? Would it probably do good or harm? 
Please Comment. 



QUESTIONNAIRE EVALUATION 



97-0 RESPONDENT'S EVALUATION 
97-1 RESPONDENT'S EVALUATION 
97-2 RESPONDENT'S EVALUATION 
97-3 RESPONDENT’S EVALUATION 
97-4 RE S P ON DENT ’ S E VALUAT I ON 



OF 


QUESTIONNAIRE: 


EXCELLENT 


OF 


QUESTIONNAIRE: 


GOOD 


OF 


QUESTIONNAIRE: 


MODAL 


OF 


QUESTIONNAIRE : 


POOR 


OF 


QUESTIONNAIRE: 


BAD 



[WRITTEN COMMENT ON THIS QUESTIONNAIRE IS WELCOMED:] 



51 
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PART III 



QUESTIONS PERTAINING TO MAN-MACHINE GAMES OR 
SIMULATIONS AND MANUAL GAMES 




55 
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CHARACTERIZATION GENERAL DESCRIPTION 



#98. Control Team: By the phrase control team we mean a formal team 

as part of the game making up rules or interacting with the other teams 
as the game progresses (#98-3). This should be contrasted with game 
management control(#98-2) where the directors or the managers or referees 
do not play a direct, important, game-influencing role. For example, 
most business games under this categorization do not have a control 
team. Few two-sided dueling games have control teams. Almost all 
political-military exercises have control teams. 



98-0 


CONTROL 


TEAM .‘UNKNOWN 


98-1 


CONTROL 


TEAM: NO 


98-2 


CONTROL 


TEAM:YES, BUT COULD BE COMPUTERIZED 


98-3 


CONTROL 


TEAM: YES, MUST BE USED 



Low Middle High 

I I $ I 

Confidence Level 



#99. Number of Live Player Teams: This excludes a control team. If 

the game has been designed to have a variable number of teams, this 
should be noted. The number of live teams actually used in different 
runs should also be indicated. 

99-0 NUMBER OF LIVE PLAYER TEAMS: ( ) Specify 



#100. Number of Robots: The same observations hold for robots or 

completely computerized teams. In simulations where a combat system is 
being simulated, such as in a totally computerized duel, we may regard 
the model as having two robots, one for each side playing the other. 

A simple test for this classification would to ask how to convert 
this game into a man-machine game. In order to do so, some of the 
automated decisionmaking of one or both sides would be removed and 
replaced by live player decisions. 

100-0 NUMBER OF ROBOT TEAMS: ( ) Specify 
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#101. Sequencing of Moves: There are some games (such as many of the 

war games) played where moves are simultaneous. Furthermore, many games, 
such as two-person matrix experiments, usually utilize simultaneous moves. 
There are other games in which the moves are in fixed sequential order; 
examples of such are chess or checkers. There are other games in which 
the moves are in variable order where frequently either chance will determine 
the next move or a player is in a position to give the move to another 
player. Craps is an example of just such a game; depending on how one 
defines chess, one pauses to see who selects sides at the beginning in the 
first move in an invariable order after which it is in fixed sequential 
order. Another set of examples are sporting events. In baseball the 
batting order is fixed. In football, the inter team play goes in no 
particular order although a series of downs is in fixed format. 

In some games some of the moves may be simultaneous whereas others may 
be in variable order. For example, in some strategic war games it may 
be required to pay costs for force maintenance every period. However, when 
new weapons sytems investment considerations are included, it is up to 
the individual team to decide whether or not they intend to invest. 



101-0 SEQUENCING OF MOVES : 
101-1 SEQUENCING OF MOVES : 
101-2 SEQUENCING OF MOVES : 
101-3 SEQUENCING OF MOVES: 



UNKNOWN 
SIMULTANEOUS 
FIXED SEQUENTIAL 
VARIABLE ORDER 

Low Middle High 



I 1 I 1 

Confidence Level 



#102. Moves per Team: In this case, for complex games, we are referring 
to a move by the team as a whole, not to the many individual small transac 
tions that might be taking place. 

There are some games, such as damage exchange rate and attrition evalua- 
tion games or continuous search games, in which the simulation or the 
computation is basically a mathematical procedure with no clear 
definition of move. In this instance we classify the move description 
102 - 0 . 

102-0 MOVES PER TEAM: NA; UNKNOWN 

102-1 MOVES PER TEAM: MOVES PER TEAM ( ) Specify 
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#103. Complementary Procedures: #103-2 we include lectures; #103-4 

we include mathematical solutions such as game theoretic solutions; and 
#103-6 we note field exercises. 



103-0 


COMPLEMENTARY 


PROCEDURES 


103-1 


COMPLEMENTARY 


PROCEDURES 


103-2 


COMPLEMENTARY 


PROCEDURES 


103-3 


COMPLEMENTARY 


PROCEDURES 


103-4 


COMPLEMENTARY 


PROCEDURES 


103-5 


COMPLEMENTARY 


PROCEDURES 


103-6 


COMPLEMENTARY 


PROCEDURES 



NA; UNKNOWN 

NONE 

LECTURES 

CASE STUDIES/HISTORY 

SIMULATION/ANALYSIS 

"EXPERIENCE" 

FIELD EXERCISE 

Low Middle High 

I I I I 

Confidence Level 



104-0 


FORMAL 


GAME 


TYPE 


104-1 


FORMAL 


GAME 


TYPE 


104-2 


FORMAL 


GAME 


TYPE 


104-3 


FORMAL 


GAME 


TYPE 


104-4 


FORMAL 


GAME 


TYPE 



NA; UNKNOWN 

UNDEFINED, PAYOFFS NOT SPECIFIED 
CONSTANT SUM GAME 
NON-CONSTANT SUM GAME 
ONE-PERSON MODEL (MAXIMIZATION) 



Low Middle High 

i I f L 

Confidence Level 




Itl05. Rules: //105-1 Rigid manual. An example of a rigid manual game 

would be chess. All of the rules are well specified in advance and 
the game is a manual game. #105—2 Semi-rigid manual. An example would 
be a war game where, although the fire power and other planning factors 
are supplied, there may be some questions during the game that are 
addressed to a referring board and certain rules or rulings are made during 
the course of play. It 105-3 Free-form or referee's direction would be 
a game such as a political military exercise in which the control teams 
and the referee's direction are critical to conducting the game. The 
melding of the moves and adjudication of attempted moves is a critical 
feature of such a game. It 105-4 Rigid computerized rules are dis- 
tinguished from rigid manual inasmuch as in general the rules of manual 
games are much more visible and hence much more open to questioning 
than are those of computerized games. One of the major dangers of 
using computerized games is that a great amount of bad modeling and 
theorizing can be hidden in computer programs. #105—5 Semi-rigid 
rules computerized. In this case the game may be computerized, but 
not all of the rules are necessarily described. For example, one 
might have a business or marketing game in which although virtually 
everything to do with sales, manufacturing, and internal running of 
the firm has been computerized L press releases and newspapers are 
issued to the firm, thus adding a verbal and somewhat less formalized 
component to the game. It 105-6 In some instances games or simulations 
are nothing more than the dynamics of the behavior of a formal mathe- 
matical model or computer program. The category also includes rules 
that are well defined in a game theoretic sense. 

In some games, although all rules are given they are so complicated that 
no single individual will know all of them (// 105-7). In other games 
part of the purpose is to discover unstated rules ( #105—8 ) . In free- 
form games, there may be considerable uncertainty concerning basic 
structure (// 105-9). 



105-0 


RULES : 


NA; UNKNOWN 


105-1 


RULES : 


RIGID MANUAL 


105-2 


RULES : 


SEMI-RIGID MANUAL 


105-3 


RULES : 


FREE FORM/REFEREE'S DIRECTION 


105-4 


RULES : 


RIGID COMPUTERIZED 


105-5 


RULES : 


SEMI-RIGID COMPUTERIZED 


105-6 


RULES : 


PROGRAM OR FORMAL MATH MODEL 


105-7 


RULES : 


WELL DEFINED BUT TOO LARGE FOR COMPREHENSION 


105-8 


RULES : 


SOME RULES, NOT KNOWN 


105-9 


RULES : 


UNCERTAINTY CONCERNING BASIC STRUCTURE 



Low Middle High 

I 1 l I 

Confidence Level 



* 
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#106. Structure of Game: In the category structure of games we include, 

under #106—3 matrix game, a game which could conceivably be approximated 
by a matrix even though in some cases the strategies may be continuous. 

For example, it may be permitted to have a player pick any price he 
wants in the range from $0 - $100, but in general he may be limited to 
bids in units of a dollar. Even if he wanted to make it a continuous 
game, it is quite possible that the machine would round it off by a 
finite approximation and thereby make it a matrix game. 

Some games may fit into more than one category. For example, a business 
game may have a formal structure such as an iterated matrix game; however, 
it begins play with a scenario describing the state of the market. 

#10 6—2 Explicit mathematical 2x2 matrix games. #106-3 Other matrix 
game. Here we are referring to a purely abstract mathematical structure 
provided as the venue for the game. #106-4 Implicit computerized damage 
exchange calculation which in fact can be regarded as a computation on 
an enormously large matrix. #106-5 Iterated matrix game. In many 
instances, such as many plays with the Prisoner 1 s Dilemma, the game is 
played in a dynamic mode. However, a great amount of the decisionmaking 
is performed on the same structure period after period. Many business 
games have the same "bat tlefield" of a more or less similar market each 
period. #106-6 Free form extensive . Games like political military 
exercises would fall in this category. They are played move by move, 
but they are not necessarily repeating the same situation on each move. 

#106-7 Formal extensive. A game such as chess is played in formal 
extensive manner. The rules are rigid and well-defined; however, the 
players move play by play and do not enunciate overall strategies for 
this game. The remaining two categories concern games in characteristic 
function form or games in which the prime area of investigation is coalitions. 
#106-8 Free form characteristic function. Such a game would be one 
devoted to studying the coalition possibilities for a treaty on the use 
of the Danube, as just one example. #106-9 Formal characteristics 
function form. There have been experiments done using games specified 
in characteristic function form where the experimenter studies how the 
players divide the money they receive from cooperative acts. 



Question #106 calls for some familiarity with several concepts of game 
theory. If you are not sufficiently familiar with the terms to answer, 
check #106-11 and proceed to the next question. 



106-0 


STRUCTURE 


OF 


GAME: 


106-1 


STRUCTURE 


OF 


GAME: 


106-2 


STRUCTURE 


OF 


GAME: 


106-3 


STRUCTURE 


OF 


GAME: 


106-4 


STRUCTURE 


OF 


GAME: 


106-5 


STRUCTURE 


OF 


GAME: 


106-6 


STRUCTURE 


OF 


GAME: 


106-7 


STRUCTURE 


OF 


GAME: 


106-8 


STRUCTURE 


OF 


GAME: 


106-9 


STRUCTURE 


OF 


GAME: 


106-10 


STRUCTURE 


OF 


GAME: 


106-11 


STRUCTURE 


OF 


GAME: 



UNKN0WN/NA 

SCENARIO/VERBAL DESCRIPTION 

2X2 MATRIX 

OTHER MATRIX 

IMPLICIT MATRIX 

ITERATED MATRIX 

"FREE FORM EXTENSIVE” 

FORMAL EXTENSIVE 

n FREE FORM” CHARACTERISTIC 

FORMAL CHARACTERISTIC FUNCTION 

PURE MATH MODEL: SIMULATED OR ANALYZED 

QUESTION NOT CLEAR Low Mi ddle 



High 



L 



X 



X 



Confidence Level 



#107. Information State: #107-1 and #107-2. The first refers to perfect 

information in the game theoretic sense: all players know all that can 

be known at all times . An example of a game with perfect information 
is a chess game. Incomplete information is the situation that prevails 
in a poker game. The kibitzers or a referee may know what the hands of 
all the players look like, but the players do not know each other's hands. 
#107-5 Considerable structural uncertainty refers to games in which the 
rules and the general environment are by no means clear at the beginning 
of the game. An example would be an extremely free form political game 
to be played in a future time period with the skimpiest of scenarios 
available. #107-3, #107-6 In some games that are designed to teach 
procedures and approaches to a problem, it is worth distinguishing 
information states in which information can be bought as contrasted with 
those in which calculation can be bought. For example, in a business 
game one may have information concerning the reaction of the market to 
various arrays of prices. On the other hand, one may not have procedures 
for fitting models to this information. The procedures such as least 
square statistical packages can be regarded as calculation packages. In 
some cases these may be available to players from "consultants" who charge 
for their use. 



107-0 


INFORMATION 


STATE: 


NA; UNKNOWN 




10 7-1 


INFORMATION 


STATE: 


PERFECT 




107-2 


INFORMATION 


STATE: 


INCOMPLETE 




107-3 


INFORMATION 


STATE: 


INFORMATION 


CAN BE BOUGHT 


10 7-4 


INFORMATION 


STATE: 


SOME RULES NOT KNOWN/MIXED 


107-5 


INFORMATION 


STATE: 


CONSIDERABLE 


STRUCTURAL UNCERTAINTY 


10 7-6 


INFORMATION 


STATE: 


CALCULATION 


CAN BE BOUGHT 



Low Middle 



High 



Confidence Level * 

#108. Computer Use: Under #108-1 bookkeeping/ light staff work, we include 

the use of the computer for somewhat more than straight bookkeeping, but 
somewhat less than one might wish to describe as heavy analysis. #108-6 
Man-machine on-line interrogative mode. We distinguish this from man- 
machine interactive in the sense that interactive merely implies that the 
machine does the computations on the environment, whereas interrogative 
implies that in the process of calculating^ the machine questions the 
player and obtains answers from the player. 



108-0 COMPUTER USE: 
108-1 COMPUTER USE: 
108-2 COMPUTER USE: 
108-3 COMPUTER USE: 
108-4 COMPUTER USE: 
108-5 COMPUTER USE: 
108-6 COMPUTER USE: 
108-7 COMPUTER USE: 



NONE/BOARD/ FIELD, ETC. 

BOOKKEEPING/LIGHT STAFF WORK 
ANALYTICAL AID TO PLAY 

ANALYTICAL AID DEBRIEFING/POST GAME ANALYSIS 
ANALOGUE 

MAN-MACHINE INTERACTIVE 

MAN -MACHINE ON-LINE (INTERROGATIVE MODE) 

OTHER ( ) Specify 



Low Middle High 

1 1 1 

Confidence Level 



L 
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//109 Gaming Facilities: #109-7 This refers to the situation where a 

special system set of languages or program may have been written to 
accompany the running and general handling of a specific game. For 
example, some games run at the labs at Berkeley and some games run with 
the TRACE system at SDC or at UCLA depend upon the availability of 
much of the specialized extra hardware and software. Although it 
is possible that the games themselves can travel, much of their power 
is lost when the accompanying programs and equipment are not available. 



109-0 


GAMING FACILITIES: 


NA; UNKNOWN 


109-1 


GAMING FACILITIES: 


SPECIAL BUILDING OR INSTITUTION 


109-2 


GAMING FACILITIES: 


SPECIAL LAB 


109-3 


GAMING FACILITIES: 


DEDICATED COMPUTER 


109-4 


GAMING FACILITIES: 


RENTED LAB 


109-5 


GAMING FACILITIES: 


RENTED "SPACE" 


109-6 


GAMING FACILITIES: 


TEMPORARY "FREE SPACE"; INFORMAL 


109-7 


GAMING FACILITIES: 


SPECIAL LANGUAGE, LIBRARY OR COMPUTATIONAL SYSTEM 



Low Middle High 

4 1 * 1 

Confidence Level 

GAME OPERATION TIME FRAME 



110-0 SET UP TIME ( ) Specify 



//111. Elapsed time of run, start to finish: We refer to the playing time 

of a single game or, in the case of experimental games, of a single experi- 
ment which could involve several replications; for example, when a series 
of experiments is run sequentially over several days. In some instances 
the nature of the game and its format make this figure quite precise. 

In other cases there is a variability of several orders of magnitude 
concerning how long the game takes to play. For example, some business 
games are run on the basis of one decision a day or one decision a week 
that is made in less than an hour; otherwise, the individuals carry 
on their normal routines. 

Under this category we refer to total time lapsed from the start of the 
game to the debriefing. In some informal instances, such as running a 
game with a class, one may run the game for the whole of a semester. For 
example, the Carnegie Tech game may run for a period of seven or eight 
months. The games of the Studies, Analysis and Gaming Agency are frequently 
run in two different modes; One is a 3-1/2 day intensive game, and the 
other stretches over several weeks. In the case of experimental games, 
games are often run in parallel — possibly intensively during one evening 
for the whole of a game, but in some cases batches of games are run compri- 
sing an experiment. In other cases players may run for more than one day. 

If a game is run on more than one mode, the different elapsed times should 
be indicated. 

111-0 ELAPSED TIME OF RUN, START TO FINISH: ( ) Specify 

Low Middle High 

I I ! _i 

Confidence Level 
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#112. Player Game Play Time: By this we mean the amount of time spent 

in actually playing a game. This includes briefing, decisionmaking, and 
debriefing associated with the game. 



112-0 PLAYER 
112-1 PLAYER 
112-2 PLAYER 
112-3 PLAYER 
112-4 PLAYER 
112-5 PLAYER 
112-6 PLAYER 



GAME PLAY TIME: 
GAME PLAY TIME: 
GAME PLAY TIME: 
GAME PLAY TIME: 
GAME PLAY TIME: 
GAME PLAY TIME: 
GAME PLAY TIME: 



NA; UNKNOWN 
<3 HOURS 
>3-6 HOURS 
>6-12 HOURS 
>12-24 HOURS 
>1-7 DAYS 
>1 WEEK ( 



) [Specify] 

Low Middle High 



Confidence Level 



//113. Formal Game Prebriefing Procedure: We note that the military use 

the word "indoctrination" when describing materials sent out prior to 
the formal briefing time in a game. 



The distinction to be made here concerns whether or not a game has a 
formal prebriefing procedure or whether the prebriefing is informal or 
nonexistent. For example, chess players in general need no prebriefing 
if they already know the rules. Some simple games such as experiments 
with 2x2 matrix games, may be run with an informal briefing from the 
experimenter who has either decided to dispense with formal control or 
has overlooked the use of formal control in the verbal description of 
the game. 

113-0 FORMAL GAME PREBRIEFING PROCEDURE: NA; UNKNOWN 

113-1 FORMAL GAME PREBRIEFING PROCEDURE: NO 

113-2 FORMAL GAME PREBRIEFING PROCEDURE: YES 



Low Middle High 



l | i 

Confidence Level 
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#114. Formal Briefing Time (%) : This can be described as a percentage 

of player game play time. For example, if it takes a player ten hours 
to play a game and there is a briefing session of 1/2 an hour, this 
means that briefing time is five percent of game play time. 

114-0 FORMAL BRIEFING TIME (%) : ( ) [Specify] 

Low Middle High 

s i i 1 

Confidence Level 



#115. Debriefing Time: This is also stated as a percentage of game 

play time for the player. In general, many experimental games, and 
certainly games for entertainment, have little if any debriefing. 
Occasionally there are post mortems after chess. Operational games and 
games for teaching and training may have considerable debriefing. For 
operational use, the length of debriefing is fairly clear f for example, 
after a SAGA game, there may be a half day set aside (beyond the three 
days of play) specifically for discussion and formal debriefing. How- 
ever, with a game used for teaching purposes, such as the Carnegie Tech 
game, one might regard the complete course taken with the game as a 
briefing-debriefing session; in which case, one could claim that the 
debriefing and briefing time could easily be as large if not larger than 
the game-playing time. 

If a game is used for different purposes one should indicate the briefing 
and debriefing time of each. The context of purposes stated in the earlier 
question should make clear the specific category to which a game belongs. 

115-0 DEBRIEFING TIME (%) : >25 ( ) [Specify] 

Low Middle High 

j i i i 

Confidence Level 
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// 116. Control Time, Total Expenditure: This includes briefing time, 

running time, and debriefing time of the control group. For example, 
in a SAGA exercise, there may be field trips and so forth before the 
games scenario can be written. This type of work would be classified 
under game construction. The time we are interested in here is that 
amount spent by the individuals composing the control team for running 
purposes. If it is necessary for members of the control team to be 
briefed or indoctrinated for several weeks in advance, this would be 
counted as part of control time* If, however, as is usually the case, 
they join the group merely a day or two or even less before play time, 
we would start to count control team time from this point. In some 
instances there is not a formal control team; however, there is never- 
theless a game director — formal or informal — whose time is being 
used to supervise the process. This should also be counted even though 
this will generally amount to no more than a few hours or a few days . 

116-0 CONTROL TIME, TOTAL EXPENDITURE: MAN WEEKS ( ) 

[Specify] 

Low Middle High 



i 1 1 i 

Confidence Level 



#117 . Post Debriefing Analysis (Intensity): Thi 5 > refers to the 

analysis of the game run after the game is over; the debriefing may 
have taken place. In other words, this should not be confused with 
debriefing. It refers to the analysis which may be done by researchers, 
possibly the players in a different mode, or others to determine what 
has been learned from the game. In the case of experiments, this is 
quite obviously where much of the work is concentrated. In the case of 
operational games, this is where much of the work should be concentrated 
if one wishes to measure the effectiveness of the exercise. it 117-0 

Post debriefing analysis. For straight simulations there is no debrief- 
ing, hence this category is not applicable. At the same time simulations 
invariably involve analysis after they are run; this is picked up in 
#118. Under #117—4 Considerable, would be where the analysis time 
may be even more than the game-playing time. 



117-0 


POST 


DEBRIEFING 


117-1 


POST 


DEBRIEFING 


117-2 


POST 


DEBRIEFING 


117-3 


POST 


DEBRIEFING 


117-4 


POST 


DEBRIEFING 



ANALYSIS (INTENSITY) 
ANALYSIS (INTENSITY) 
ANALYSIS (INTENSITY) 
ANALYSIS (INTENSITY) 
ANALYSIS (INTENSITY) 



NA ; UNKNOWN 

NONE 

SLIGHT 

MODERATE 

CONSIDERABLE 

Low Middle High 



i i 

Confidence Level 



ERIC 



i 



j 



r 



T 



r 



1 



1 
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#118. Analysis Time: Answer this question in terms of man-weeks, or 

the percentage of actual time spent by individuals involved in the 
analysis as compared with total game play time. There may be an enormous 
amount of automated analysis going on with a small expenditure of human 
time. This question is concerned with the human time. 

118-0 ANALYSIS TIME: ( ) [Specify] 



119-0 SET UP COSTS AS % OF TOTAL COSTS TO OPERATE: ( 



[Specify] 

Low Middle High 



Confidence Level 



#120. Cost to Operate at New Location: Except for going to the new 

location we assume that all other costs will allocate in the same way. 

In other words, if one is calculating on free secretarial help at one 
place, one calculates some free secretarial help at the other place. 

It makes a great difference whether or not you can bring in an operating 
crew. However this means that cost to operate should be looked at as 
the minimum cost to bring in an operating crew or cost to train new 
people on location. If no crew is available, it may be either impos- 
sible to transfer the game or inordinately expensive. 

When we refer to new location we assume that the new location has hard- 
ware that is suited for the game involved. 

120-0 COST TO OPERATE AT NEW LOCATION: ( ) [Specify] 

Low Middle High 



Confidence Level 




GG 
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GAME .CHARACTERIZATION 
PLAYERS 



#121. Player Selection. #121-5 "Proprietary interest advocates" refers 
to the use of players who have a personal interest in the actual use 
of the game or in the use of the results of the game for some specific 
purpose. For example, a proprietary interest advocate might be a group 
of individuals either advocates for or opponents against a weapons 
system, such as MIRV, or the SST, or a specific place of hardware. 

The game may be used as part of an ongoing advocacy process. In cases 
such as this, it is extremely important to sort out players whose 
play surrounds the environment of the game from players whose interests 
cease with the use of the game for whatever its explicit stated purposes. 



121-0 PLAYER SELECTION: 
121-1 PLAYER SELECTION: 
121-2 PLAYER SELECTION: 
121-3 PLAYER SELECTION: 
121-4 PLAYER SELECTION: 
121-5 PLAYER SELECTION: 



UNKNOWN 

UNPAID INDIVIDUAL VOLUNTEER 
VOLUNTARY GROUP 
PAID VOLUNTEER 
COURSE REQUIREMENT 
"PROPRIETARY INTEREST ADVOCATES" 



Low Middle 



High 



I I l 

Confidence Level 



#122. Player Characterization: Post graduate refers to individuals 

at a war college or other academic institution. #122—3 Professional 
refers to in the sense of the game being played. A military man 
playing a war game would be regarded as a professional. If he were 
playing a business game, he would notjin general, be regarded as a 
professional. 



122-0 PLAYER CHARACTERIZATION: 
122-1 PLAYER CHARACTERIZATION: 
122-2 PLAYER CHARACTERIZATION: 
122-3 PLAYER CHARACTERIZATION: 
122-4 PLAYER CHARACTERIZATION: 



UNKNOWN 

POST GRADUATE 

ADULT (NON PROFESSIONAL) 

PROFESSIONAL 

OTHER 



Low Middle 



High 



i _ —i i 

Confidence Level 



O 

ERIC 



• 
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#123. Player Use. In some man-machine exercises live players are used 
only because they are cheaper or more readily available than a simu- 
lated player. There is no attempt to train them nor are their goals of 
particular concern to the exercise. In this sense they are merely a 
substitute for machinery; this possibility is described in # 123-0. 

123-0 PLAYER USE: ONLY AS "MACHINERY" 

123-1 PLAYER USE: AS PLAYERS 

[IF ANSWER TO #123 IS #123-0 SKIP THE REMAINING QUESTIONS ON PLAYERS 
AND START ON "GAME USE" QUESTIONS] 



124-0 


PLAYER 


PAYOFFS : 


UNKNOWN 


124-1 


PLAYER 


PAYOFFS : 


MONEY WAGE 


124-2 


PLAYER 


PAYOFFS : 


GRADES OR PAYMENT 


124-3 


PLAYER 


PAYOFFS : 


FIXED PRIZE 


124-4 


PLAYER 


PAYOFFS : 


PRIZE PROPORTIONAL TO PERFORMANCE 


124-5 


PLAYER 


PAYOFFS : 


"EDUCATION" 


124-6 


PLAYER 


PAYOFFS : 


NOT SPECIFIED 


#125. 


Player 


Pretest 


Comprehension Test: Is a check made 



the players fully comprehend the game prior to play? This 
the use of a practice play followed by questions. 



to see if 
includes 



125-0 PLAYER PRETEST COMPREHENSION TEST : UNKNOWN 

125-1 PLAYER PRETEST COMPREHENSION TEST : NO 

125-2 PLAYER PRETEST COMPREHENSION TEST : YES 

Low Middle High 



| l 

Confidence Level 



L 
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#126. Player Pretest: #126-1 Refers to the case where no particular 

pretesting after the selection of the players has been performed; 

#126-2 refers to games in which one has in fact run pretests on the 
players. This may be a California Personality Inventory, it may be 
IQ-test s, and so forth. One runs subjects or players through a bar- 
rage of tests outside of the formal game. 

Player pretest could be a test for IQ, general knowledge, etc., or 
some sort of personality test. It does not necessarily imply a compre- 
hension test for the game. 

126-0 PLAYER PRETEST: UNKNOWN 

126-1 PLAYER PRETEST: NO 

126-2 PLAYER PRETEST: YES ( ) [Describe] 

Low Middle High 

i i l i 

Confidence Level 



127-0 PLAYER POST PLAY COMPREHENSION CHECK: NA; UNKNOWN 

127-1 PLAYER POST PLAY COMPREHENSION CHECK: NO 

127-2 PLAYER POST PLAY COMPREHENSION CHECK: YES 

Low Middle High 

i 1 1 1 

Confidence Level 
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//128. Player Perception of Success of Purpose: In this characteri- 

zation we are not asking the question as to whether the game was 
effective, but as to how the experience was perceived by the players. 

A good question that must be asked of all games is, "How does enjoyment 
correlate with the value of a game?" It is conjectured by us that up 
to a certain level there is probably a positive correlation between 
the effectiveness of a game and the level of enjoyment or enthusiasm. 
Beyond a certain level, however, we suspect that the correlation weakens 
or goes negative. A highly enjoyable game may in fact have been 
enjoyed as a game and not as an operational, research, or teaching 
device. Informally we have observed that apparently there is not a 
great amount of correlation between highly popular lecturing, acting 
performances, and the amount of information that is conveyed to the 
students. In some cases player reaction may be mixed. You may wish 
to give a percentage breakdown next to the categories or draw a small 



graph. 
















128-0 


PLAYER 


PERCEPTION 


OF 


SUCCESS 


OF 


PURPOSE : 


NOT RELEVANT; UNKNOWN 


128-1 


PLAYER 


PERCEPTION 


OF 


SUCCESS 


OF 


PURPOSE: 


HIGHLY POSITIVE 


128-2 


PLAYER 


PERCEPTION 


OF 


SUCCESS 


OF 


PURPOSE: 


POSITIVE 


128-3 


PLAYER 


PERCEPTION 


OF 


SUCCESS 


OF 


PURPOSE: 


NEUTRAL 


128-4 


PLAYER 


PERCEPTION 


OF 


SUCCESS 


OF 


PURPOSE : 


NEGATIVE 


128-5 


PLAYER 


PERCEPTION 


OF 


SUCCESS 


OF 


PURPOSE: 


HIGHLY NEGATIVE 



Low Middle High 



i i i 

Confidence Level 



129-0 


"AVERAGE" 


PLAYER'S 


ENJOYMENT : 


NOT RELEVANT; UNKNOWN 


129-1 


"AVERAGE" 


PLAYER'S 


ENJOYMENT : 


HIGHLY POSITIVE 


129-2 


"AVERAGE" 


PLAYER'S 


ENJOYMENT: 


POSITIVE 


129-3 


"AVERAGE" 


PLAYER'S 


ENJOYMENT: 


NEUTRAL 


129-4 


"AVERAGE" 


PLAYER'S 


ENJOYMENT : 


NEGATIVE 


129-5 


"AVERAGE" 


PLAYER'S 


ENJOYMENT : 


HIGHLY NEGATIVE 
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it 130. Number of Players Per Team: Do not fill in more than the three 

most frequently used categories. 



130-0 NUMBER OF PLAYERS PER TEAM: 



( 



) [Specify] 



Low Middle High 



l i l i 

Confidence Level 



131-0 REPEATED USE OF PLAYERS: NA; UNKNOWN 
131-1 REPEATED USE OF PLAYERS: NO 
131-2 REPEATED USE OF PLAYERS: YES 



Low Middle High 



i 1 i 1 

Confidence Level 



it 132. Role Playing (Individual): #132-1 Role playing: self is the 

category which indicates that no other role playing is required of the 
players in the particular game. #132-3 Role playing: specific person 

implies for example that somebody play Stalin or Mao Tse-tung or 
Mr. Nixon. #132-4 and it 132-5 A specific organization might be something 



like General Electric; an abstract 
firm. 

132-0 ROLE PLAYING (INDIVIDUAL) : 

132-1 ROLE PLAYING (INDIVIDUAL) : 

132-2 ROLE PLAYING (INDIVIDUAL) : 

132-3 ROLE PLAYING (INDIVIDUAL) : 

132-4 ROLE PLAYING (INDIVIDUAL) : 

132-5 ROLE PLAYING (INDIVIDUAL) : 



organization would be a large business 



NOT RELEVANT /UNKNOWN 
SELF 

SPECIFIC POSITION 
SPECIFIC PERSON 

SPECIFIC ORGANIZATION/INSTITUTION 
ABSTRACT ORGANIZATION/INSTITUTION 

Low Middle High 

i i i > 

Confidence Level 



133-0 ROLE PLAYING (TEAM) 
133-1 ROLE PLAYING (TEAM) 
133-2 ROLE PLAYING (TEAM) 
133-3 ROLE PLAYING (TEAM) 
133-4 ROLE PLAYING (TEAM) 
133-5 ROLE PLAYING (TEAM) 



NOT RELEVANT /UNKNOWN 
SELF 

SPECIFIC ORGANIZATION 
ABSTRACT ORGANIZATION 
SPECIFIC INSTITUTION 
ABSTRACT INSTITUTION 

Low 



Middle 



High 



i 1 1 1 

Confidence Level 
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#134 . Importance of Unstated Purposes: This question involves the 

characterization of players and what might be described as "the game 
outside of the game." For example, in the case of a business game 
where many executives from the same firm play unaccustomed roles in a 
simulated hierarchy, there may be pressures exerted on the individuals 
as a result of their being aware that they are being watched. Even in 
experimental games, the players sometimes may decide to play "fool the 
experimenter" or "give him what he wants." This question is admittedly 
subjective, but it merits serious consideration. 



134-0 IMPORTANCE OF UNSTATED PURPOSES: 
134-1 IMPORTANCE OF UNSTATED PURPOSES: 
134-2 IMPORTANCE OF UNSTATED PURPOSES: 
134-3 IMPORTANCE OF UNSTATED PURPOSES: 
134-4 IMPORTANCE OF UNSTATED PURPOSES: 

Comments or Discussion: 



NA 

HIGH 

SOMEWHAT 

SLIGHT 

NONE T M ,,„ u . . 

Low Middle High 



i i i 

Confidence Level 



#135. Game Users: Count the number of institutions where a game, 

simulation, or a direct variant is being used. This may tend to produce 
some overestimation, yet for most purposes this is the most relevant 
figure. It gives an insight into how widespread the direct use of a 
game has been or how widespread the influence of a game has been. 

135-0 CAN. USERS: ( ) [Specify] 

Low Middle High 



i i i 

Confidence Level 
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PART IV 



VOLUNTARY ASSESSMENT 




73 



70 



For the remaining questions assume that a senior professional 
(one who really knows the business) wishes to use this MSG and 
wants your evaluation along several dimensions. 

#136. Assessment - Design and Construction: What are the strengths 

and weaknesses of this MSG's design and construction? 

Comment: 



#137. Assessment - Planning Factors and Data: What in your opinion 

are the strengths, weaknesses, and constraining features of the data 
used in the MSG? How serious are the deficiencies or weaknesses, 
if any? 



Comment : 
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#138. Assessment - Documentation: How complete and useful is the 

supporting documentation? Would it be easy for some other agency 
to use the MSG, or would the extent and quality of documentation 
make this difficult or impossible? 

Commen t : 



#139 • Assessment - Operation: Are there peculiarities of operation 

that a prospective user should be aware of? Is the MSG easy to operate 
or are there unique procedural problems that one should know about? 

Comment : 
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[For Man-Machine or Manual Games Only.] 

//140. Assessment - Post Debriefing Analysis: Are the MSG’s outputs 

easy to analyze or are they intended for use in subsequent analyses? 

Comment: 



#141. Assessment - Cost Effective: Do you think that the MSG represents 

a cost-effective way to get at the issues it addresses, or would you 
recommend alternative procedures, methods or techniques? 

Comment : 
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// 142. Assessment - Validation Criteria: What questions related to 

validation have been posed and are they clear and concise or are they 
vague, confusing or non-existent? Has much attention been given to 
validation of the MSG? 

Comment : 



//143. Assessment — Validation: Based on the criteria that were 

developed, was any validation done on the MSG? What resulted? 

Comment : 



O 
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#144- Assessment — Overall: In your opinion is the MSG of 

outstanding, average, poor, or of indeterminate quality? Would you 
commend it for future use? Unqualified acceptance? Qualified? 

Comment: 





